Although applications involving long-context inputs are crucial for the effective utilization of large language models (LLMs), they also result in increased computational costs and reduced performance. To address this challenge, we propose an efficient, training-free prompt compression method that retains key information within compressed prompts. We identify specific attention heads in transformer-based LLMs, which we designate as evaluator heads, that are capable of selecting tokens in long inputs that are most significant for inference. Building on this discovery, we develop EHPC, an Evaluator Head-based Prompt Compression method, which enables LLMs to rapidly skim through'' input prompts by leveraging only the first few layers with evaluator heads during the pre-filling stage, subsequently passing only the important tokens to the model for inference. EHPC achieves state-of-the-art results across two mainstream benchmarks: prompt compression and long-context inference acceleration. Consequently, it effectively improves performance with the reduced costs associated with commercial API calls compared to prompt compressing methods. We further demonstrate that EHPC attains competitive results compared to key-value cache-based acceleration methods, thereby highlighting its potential to enhance the efficiency of LLMs for long-context tasks.

Large language models (LLMs) have exhibited exceptional capabilities in a variety of real-world tasks and applications, with an increasing need for processing long inputs in areas such as literary novels, legal documents, instruction manuals, and code documentation. Inference tasks that requires understanding of long contexts, such as long document summarization (Zhang et al., 2024), reasoning (Fei et al., 2024a), and autonomous agents (Singh et al., 2024; Chen et al., 2024), are of particular importance due to the high stakes in these scenarios. However, the deployment of LLMs is challenged by the computational and memory demands inherent to transformer-based architectures, resulting in increased latency, particularly when processing lengthy input prompts. Prompt compression, which entails substituting the input prompts provided to a language Table 1: Overall comparison of the proposed model with more succinct versions, has surfaced method in terms of average performance and latency as a promising strategy for enhancing long-text on the LongBench dataset, under the constraint understanding and mitigating associated costs. of a compressed prompt length of 2048 tokens. Current mainstream methods, such as Select-For comprehensive results, please see Table 4 Context (Li et al., 2023), LLMLingua (Jiang and Table 5. et al., 2023a) and LongLLMLingua (Jiang et al., 2023b), typically rely on pre-trained LLMs, utilizing the logits or perplexity of the prompts